Visible Cohesion: A Comparison of Reference Tracking in Sign, Speech, and Co-Speech Gesture

نویسندگان

  • Pamela Perniss
  • Asli Özyürek
چکیده

Establishing and maintaining reference is a crucial part of discourse. In spoken languages, differential linguistic devices mark referents occurring in different referential contexts, that is, introduction, maintenance, and re-introduction contexts. Speakers using gestures as well as users of sign languages have also been shown to mark referents differentially depending on the referential context. This article investigates the modality-specific contribution of the visual modality in marking referential context by providing a direct comparison between sign language (German Sign Language; DGS) and co-speech gesture with speech (German) in elicited narratives. Across all forms of expression, we find that referents in subject position are referred to with more marking material in re-introduction contexts compared to maintenance contexts. Furthermore, we find that spatial modification is used as a modality-specific strategy in both DGS and German co-speech gesture, and that the configuration of referent locations in sign space and gesture space corresponds in an iconic and consistent way to the locations of referents in the narrated event. However, we find that spatial modification is used in different ways for marking re-introduction and maintenance contexts in DGS and German co-speech gesture. The findings are discussed in relation to the unique contribution of the visual modality to reference tracking in discourse when it is used in a unimodal system with full linguistic structure (i.e., as in sign) versus in a bimodal system that is a composite of speech and gesture.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Applying mean shift and motion detection approaches to hand tracking in sign language

Hand gesture recognition is very important to communicate in sign language. In this paper, an effective object tracking and hand gesture recognition method is proposed. This method is combination of two well-known approaches, the mean shift and the motion detection algorithm. The mean shift algorithm can track objects based on the color, then when hand passes the face occlusion happens. Several...

متن کامل

Perception and Synthesis of Biologically Plausible Motion: From Human Physiology to Virtual Reality

Temporal measures of hand and speech coordination during French cued speech production p. 13 Using signing space as a representation for sign language processing p. 25 Spatialised semantic relations in French sign language : toward a computational modelling p. 37 Automatic generation of German sign language glosses from German words p. 49 French sign language processing : verb agreement p. 53 R...

متن کامل

What Does It Mean to Compare Language and Gesture? Modalities and Contrasts

I n this paper I would like to reexamine some of the traditional dichotomies between language and gesture. In order to do so, it will be necessary to consider a three-way contrast—spoken languages, signed languages, and gesture. Without this three-way comparison, we risk collapsing contrasts between visual and auditory media with contrasts between linguistic structure and co-linguistic gestural...

متن کامل

Phoneme Classification Using Temporal Tracking of Speech Clusters in Spectro-temporal Domain

This article presents a new feature extraction technique based on the temporal tracking of clusters in spectro-temporal features space. In the proposed method, auditory cortical outputs were clustered. The attributes of speech clusters were extracted as secondary features. However, the shape and position of speech clusters change during the time. The clusters temporally tracked and temporal tra...

متن کامل

On the temporal domain of co-speech gestures: syllable, phrase or talk spurt?

This study explores the use of automatic methods to detect and extract hand gesture movement co-occuring with speech. Two spontaneous dyadic dialogues were analyzed using 3D motion-capture techniques to track hand movement. Automatic speech/non-speech detection was performed on the dialogues resulting in a series of connected talk spurts for each speaker. Temporal synchrony of onset and offset ...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:
  • Topics in cognitive science

دوره 7 1  شماره 

صفحات  -

تاریخ انتشار 2015